🎯 By the end of this lecture you will be able to…
- Distinguish data collection, analysis, and interpretation — and why conflating them is dangerous.
- Choose appropriate data-collection methods and sources for a computing problem.
- Identify the instrumentation (tools, benchmarks, metrics) a study requires.
- Avoid the classic evidence errors: bad sampling, over-claiming, and correlation-as-causation.
In Lecture 3 you chose an approach. That approach now has to produce evidence. Three distinct activities turn a plan into a finding: collect the data, analyse it, then interpret what it means. They are not the same thing — and the gap between the last two is where most research goes wrong.
1Data collection
Data collection is the systematic gathering of the observations you'll reason about. In computing, "data" ranges from benchmark timings to survey responses to system logs.
Primary vs. secondary data
| Primary data | Secondary data | |
|---|---|---|
| Definition | You collect it yourself, first-hand | Collected by others, reused by you |
| Computing example | Latency you measured; a survey you ran | Public datasets (ImageNet, MNIST), GitHub archives, Kaggle |
| Pros | Fits your exact question; you control quality | Fast, cheap, large, comparable to others |
| Cons | Time-consuming; needs instrumentation | May not fit; unknown biases; licensing/ethics |
Common collection methods in computing research
- Benchmarking / measurement: instrument code to record time, memory, throughput, energy.
- Logging & telemetry: capture events from a running system.
- Public datasets: reuse standard corpora for comparability with prior work.
- Surveys & questionnaires: gather developer/user opinions at scale.
- Interviews & observation: deep, qualitative insight into practice.
- Web scraping / APIs: collect data from online sources (mind terms of service and ethics).
You rarely measure everything; you measure a sample and generalise. If the sample is biased, so is every conclusion. Testing your recommender only on power-users, or benchmarking only on tiny inputs, produces confident but wrong claims. Ask: does my sample represent the population I want to generalise to?
Running a benchmark once and reporting the number ignores variance from caching, background processes, and thermal throttling. Fix: repeat runs (e.g., 30×), discard warm-up, and report a central value with its spread (mean ± std, or median + IQR).
2Instrumentation — the tools of evidence
Instrumentation means everything you use to obtain and record data reliably: the measuring apparatus of your study. In a lab it's oscilloscopes; in computing it's a stack of software and defined metrics.
| Instrumentation type | Examples | Used to capture… |
|---|---|---|
| Profilers & timers | perf, gprof, time, JMH | execution time, CPU, memory |
| Benchmark suites | SPEC, TPC, MLPerf, GLUE | standardised, comparable workloads |
| Monitoring / logging | Prometheus, Grafana, ELK | system behaviour over time |
| Survey platforms | Google Forms, Qualtrics | human responses |
| Statistical / analysis tools | Python (pandas, SciPy), R | analysis & visualisation |
| Defined metrics | accuracy, F1, latency p95, throughput | the quantities you actually report |
Deciding what counts as success after seeing results invites cherry-picking ("well, it wins on this metric"). Pre-register your primary metric. For classification, know why you're using F1 vs. accuracy; for latency, report a tail percentile (p95/p99), not just the mean — averages hide the bad experiences.
You benchmark a web server and report "average response time = 120 ms." A reviewer says that number could hide a serious problem. What are they worried about, and which single metric would reveal it?
3Data analysis
Analysis transforms raw data into patterns: summarising, testing, and visualising. Two broad families:
- Descriptive: summarise what the data shows — mean, median, standard deviation, distributions, charts.
- Inferential: decide whether an observed effect is real or chance — hypothesis tests, p-values, confidence intervals, effect sizes.
Model A averages 91.2% accuracy, Model B 91.8%. Is B really better? Descriptive stats say B is higher. Inferential analysis (e.g., a paired test across cross-validation folds) tells you whether a 0.6-point gap is statistically significant or just noise. Without the second step, you might crown a winner that isn't one.
A good chart reveals structure a table hides (distributions, outliers, trends). A dishonest chart — truncated y-axis, missing error bars — misleads. Always show the spread of your data, not just the headline number.
4Interpretation — what does it mean?
Interpretation is where you explain the significance of your results: connecting numbers back to the research question, comparing with prior work, and stating what we now know. Analysis says "B is 0.6% higher, p = 0.03." Interpretation says "B's small but significant gain likely comes from its data augmentation, suggesting augmentation matters more than architecture for this task."
Your logs show users who use feature X churn less. Does X cause retention? Maybe — or maybe already-loyal users happen to explore more features. Observational data shows association; causal claims need controlled experiments (e.g., an A/B test). Over-claiming causation is one of the most common — and most penalised — errors in student research.
"Our method beats the baseline on MNIST, therefore it's the best image classifier." One dataset, one domain, one setting cannot support a universal claim. Interpret within your scope (Lecture 3). Honest limits build credibility; overreach destroys it.
Label each statement as analysis or interpretation: (a) "The median query time was 44 ms." (b) "The 3× slowdown on joins suggests the index isn't being used." (c) "Accuracy differences were not significant (p = 0.4)."
✅ Check your understanding
🧠 Key takeaways
- Collection → analysis → interpretation are three distinct steps; the biggest errors hide in the last one.
- Know your data source: primary (you gathered it) vs. secondary (reused) — and watch for sampling bias.
- Instrumentation = the tools and metrics that record evidence; choose metrics before running experiments.
- Analysis is descriptive and inferential — a higher number isn't "better" until significance says so; always show spread.
- Interpret within scope: never confuse correlation with causation, and never over-generalise beyond your evidence.
You can now find a problem, judge its quality, scope it, set objectives, pick an approach, and gather trustworthy evidence. Unit II goes deeper into research methods specific to computing science — models of argument, proof methods, deduction vs. induction, and performance evaluation.
Research Methodology in CS (CSEG3060) · Unit I · Lecture 4 · Dr. Mohsin Furkh Dar · UPES Dehradun